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Abstract 


Like other universities, RMIT recognises the significance of graduates’ ratings of their experience and 
has had a long-term commitment to improving student learning. As at other universities, RMIT’s 
standard subject-level survey (the Course Experience Survey [CES]) incorporates items from the 
national Course Experience Questionnaire, with the aim of eliciting student views and prompting 
improvements that will take effect before the students graduate. The university has been seeking 
strategies to make the results of these surveys more accessible to academic staff, so that staff can use 
them as a starting point for change. 

The current project is part of this work. The starting point is discipline-based analysis of the 
university’s CES data. Surveys were stratified into fifty disciplines, and categorical factor analysis 
applied to ascertain common interpretable factors. The results have been presented to staff for 
discussion in the context of planning for improvement. 

This article explores the results of the factor analysis and its potential for providing academics with 
useful information on students’ experiences. 
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Effective teaching is more likely to be achieved by helping teachers to understand how to 
interpret research findings within their own context and circumstances, and so to identify 
the strongest influences on their own students. They will then be better able to think 
seriously about how their own practice can be enhanced in the light of the best research 
evidence currently available. (Entwistle, 2005, p. 81) 

Theoretical Background: The Significance of Disciplinary Differences 


In 1993, the Course Experience Questionnaire (CEQ) was introduced in Australia as a 
graduate survey, designed to collect data on the quality of students’ learning experiences in 
Australian universities. Since that time, while CEQ results have been used to measure 
university performance, annual data have consistently shown that they vary not by university 
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but by discipline. There has been an overall improvement in the proportion of positive 
responses to CEQ items, but differences between disciplines have persisted. 

The CEQ Good Teaching items focus primarily on the effectiveness of 
communication between academic staff and students. They were developed from work by 
Entwistle and Ramsden which demonstrated that effective communication with and by 
lecturers was strongly associated with students adopting a deep approach to their study and 
hence learning effectively (Ramsden, 1991, pp. 132, 135; Wilson et al., 1997, p. 43). These 
items were envisaged as a proxy measure of the quality of student learning. 

The pilot data collected prior to the implementation of the CEQ as a graduate survey 
indicated strong and significant differences in the responses of students from different 
disciplines (Ramsden, 1991, p. 138). The persistence of differences between disciplines can 
readily be seen by inspecting the annual CEQ data, now available on the Graduate Careers 
Australia website (GCA, 2008) (see also Patrick, 2003, and the graph at attachment 1). In 
general, social science and humanities disciplines are highly rated on the Good Teaching 
scale, whereas science and engineering disciplines are relatively poorly rated. Ramsden 
recognised this in his initial paper on the CEQ; he concluded that the differences between 
disciplines in terms of culture and resources are so marked that comparisons between 
institutions should only be made within disciplines (Ramsden, 1991, p. 139). 

Differences between disciplines in terms of approach to teaching have emerged from 
other studies. Using different items, Santhanam and Hicks (2002) analysed the differences in 
students’ opinions in evaluating their lecturers and their subjects, across two principal 
discipline areas. The items in their study asked students to rate the effectiveness of the teacher 
and the effectiveness of the curriculum. In this study, sciences/mathematics students were 
more positive about the teaching they received than were arts/humanities/social sciences 
students. Santhanam and Hicks concluded that content differences between disciplines 
influenced both the teachers’ approaches and the expectations of the students. This 
interpretation is consistent with Becher’s view of disciplinary culture; he argues that there are 
significant differences between disciplines in teaching techniques, student learning needs, and 
curriculum design (Becher, 1994). Similarly, Lueddeke argues that disciplinary differences 
have a significant influence on academics’ interest in scholarly reflection and teaching 
improvement (Lueddeke, 2003). Exploring teaching approaches, Trig well, Prosser and their 
colleagues have also found disciplinary differences: they argue that expository teaching is 
more common in the ‘hard’ sciences and a student-centred approach more common in social 
sciences and the arts (see e.g. Lindblom-Ylanne et al., 2006). 

Entwistle (2005) reports recent discipline-based investigations in the United Kingdom 
that complicate these assertions. He describes university education as a process of initiation 
into the complex culture of a particular field. While this does involve significant conceptual 
development (for which a ‘deep approach’ is particularly useful), he now sees it more broadly 
as the development of a distinctive, discipline-specific way of ‘thinking and practising’ 
(Entwistle, 2005, p. 72). 

While recognising the significance of disciplinary cultures, Knight and Trawler (2000) 
also challenge the widespread practice of focusing on the individual teacher. They argue that 
attempts to improve students’ learning experiences must take account of contextual and 
structural issues, and the changes that they have observed in university life over the past 
twenty years. In particular, they nominate intensification of work; managerialism and a loss of 
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academic autonomy; a loss of collegiality; ‘greedy’ institutions; and ageing, malaise and 
marginality among academic staff (Knight & Trowler, 2000, pp. 71-72). 

The present project is designed to explore the dimensions of student experience at 
discipline level. The aim is to provide academic staff with data that connect with the 
experiences of their own students and which will be useful to them in identifying where and 
how students’ experiences might be improved. This article focuses in particular on strategies 
for presenting the analyses generated by the project. 

Methodology 


Sample and survey 

It is university policy that each subject [locally termed a course] conducts a survey at 
least once a year. The data analysed in the study derive from the 2007 subject surveys; the 
data analysed for this article were collected from surveys conducted in semester two (July — 
October). In this period, surveys were completed in 1466 Higher Education subjects across 
the university, covering 50 discipline areas — a total of 33,156 completed survey forms. 
Many students will have completed more than one survey form: full-time enrolment 
comprises four subjects per semester, and full-time students could well be asked to complete 
four surveys in a semester. 

The survey instrument being used (the Course Experience Survey) has been locally 
developed to explore different aspects of student experience. While its statistical properties 
have not been previously evaluated, it does include the items of the CEQ Good Teaching 
scale (GTS) and other items from the CEQ. It also includes items relating to study resources 
and learning facilities. 

Table 1 lists the items in the survey, along with the labels used by the researchers. The 
Good Teaching items from the CEQ are highlighted. Responses (as with the CEQ) are on a 5- 
point Likert scale, with 1 labelled strongly disagree and 5 labelled strongly agree. 

Table 1 

Course Experience Survey items 



Course Experience Survey item 

Label 

1 

The learning objectives in this course are clear to me 

Objectives 

2 

I am learning what I expected to in this course 

Expectations 

3 

This course is well organised 

Organised 

4 

The teaching staff are extremely good at explaining things 

Explaining 

5 

The teaching staff normally give me helpful feedback on how I am going in this 
course 

Helpful feedback 

6 

This course contributes to my confidence in tackling unfamiliar problems 

Problem-solving 

7 

Assessment tasks in this course require me to demonstrate what I am learning 

Assessment 

8 

The amount of work required in this course is about right 

Workload 

9 

The teaching staff in this course motivate me to do my best work 

Motivate 

10 

I enjoy doing the work for this course 

Enjoyment 

11 

I find the learning resources for this course useful (e.g., notes, handouts, readings, 
AV materials) 

Learning resources 

12 

The web-based (online) materials for this course are effective in assisting my 
learning 

Online materials 

13 

There is effective use of other computer-based teaching materials in this course 

Computer- 

materials 




Journal of Institutional Research, 14(1), 96 - 108 . 99 




Course Experience Survey item 

Label 


14 The facilities (such as classrooms, lecture theatres, studios, labs) are adequate for Facilities 


this course 

15 I feel I can actively participate in my classes Participate 

16 There is good balance between theory and practice Theory/prac 

balance 


17 

The teaching staff work hard to make this course interesting 

Interest 

18 

I can see how I'll be able to use what I am learning in this course in my career 

Use 

19 

The staff make a real effort to understand difficulties I might be having with my 
work 

Understand 

difficulties 

20 

The staff put a lot of time into commenting on my work 

Comment 

21 

Overall, I am satisfied with the quality of this course 

Satisfaction 


Data analysis 

In previous studies, the robustness of the CEQ scales has been tested by factor 
analysis across the whole body of respondents. Implicitly, this approach assumes that the 
dimensions of students’ experience will be the same across different disciplines, and that what 
varies is the quality of their experience. By contrast, the present study uses factor analysis at 
discipline level, to distinguish the dimensions of student experience for each discipline (see 
Bedford et al., 2008). 

Because responses to the survey items were on a Likert scale, which cannot be 
assumed to be linear (Linting, Meulman, Groenen, & Van der Kooij, 2007; Meulman, Van 
der Kooij, & Heiser, 2004), the data were first transformed using categorical principal 
component analysis (CATPCA). CATPCA is a non-trivial function of SPSS™ which 
commences analysis via optimal quantification, a process whereby categories of the 
categorical variables are appointed continuous numeric values (Linting et al., 2007; Meulman 
et al., 2004). This process provides the numeric values that are required for variance and 
Pearson correlation calculations (Linting et al., 2007). Importantly, the solution is iteratively 
computed from the ordinal data, as opposed to being derived from a correlation matrix, as 
with traditional principal component analysis (PCA) (Gifi, 1990). Like traditional PC A, 
CATPCA attempts to extract factors that can account for as much variance in the variables as 
possible. Because of the transformation of the fixed values into quantified values, CATPCA 
typically accounts for more variance than PCA (Linting et al., 2007). 

Lactor analyses were undertaken separately for each of the discipline areas for which 
data were available. The extraction method used was principal component analysis, with 
varimax rotation and Kaiser normalisation. Live factors were elicited, corresponding to the 
number of factors identified in traditional principal component analysis of CEQ data (see 
Ramsden, 1991; Richardson, 1994). 


Results 


The five factor model had considerable explanatory power, accounting for around 
70% of the variation in responses. An initial factor analysis across the whole cohort of 
students resulted in the emergence of a clear Good Teaching factor comprising five of the six 
expected items. At this level, the factors which emerged were: 

• Good teaching: comment (Q20), give useful feedback (5), understand difficulties (19), 
motivate (9), interest (17) 
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• Engagement: use (18), enjoy (10), theory/prac balance (16), workload (8), problem¬ 
solving (6), participate (15), assessment (7) 

• Organisation: objectives clear (1), organised (3), expectations (2), explaining (4) 

• Resources: online (12), computer (13), learning resources (11) 

• Facilities: facilities (14) 

Analysis of variance using the established Good Teaching scale showed, as expected, strong 
and significant differences between disciplines, with mean scores ranging from 16 to 69 on a 
scale from -100 to +100. As in previous studies, engineering and business students rated their 
experience low on this scale by comparison with social science and education students. The 
results are shown for some example disciplines in Figure 1, below. 



Discipline 

Figure 1 

Student Good Teaching error bars, selected disciplines. Semester 2, 2007 

Note : The disciplines shown are accounting (ACCT), civil engineering (CIVE), communications (COMM), 
economics (ECON), marketing (MKTG), and visual art (VART).. 

The factor analyses by discipline complicate and add depth to this picture. 

While the items from the CEQ Good Teaching scale were commonly associated in a 
single factor, the significance of this factor varied between disciplines, and items associated 
with the factor also varied. In 26 of the 50 disciplines, a Good Teaching factor emerged first 
from the analysis; however it was the second factor in 16 disciplines and less salient in the 
remaining 8. 

Table 2 provides an example comparison of the two most salient factors in two 
disciplines, one from the social sciences, and the other from commerce. Mean scores on the 
standard GTS scale were relatively high for both these disciplines, as shown in Figure 1. 
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Table 2 

Example factor structures 



Communications 

N = 2573 

Marketing 

N = 1451 

Factor 1 

Good teaching 

Organised and purposeful 


Questions 20, 5, 19, 9, 17: 

Questions 3, 4, 2, 1, 17: 


20 Staff put a lot of time into commenting 

3 This course is well organised 


on my work 

4 The teaching staff are extremely good 


5 Teaching staff normally give me 

at explaining things 


helpful feedback on how I’m going 

2 I am learning what I expected to in this 


19 Staff make a real effort to understand 

course 


any difficulties I might be having with 

1 The learning objectives in this course 


my work 

are clear to me 


9 Teaching staff motivate me to do my 

17 The teaching staff work hard to make 


best work 

this course interesting 


17 The teaching staff work hard to make 
this course interesting 

Factor accounts for 19% of variance 

Factor accounts for 19% of variance 

Factor 2 

Meaningful engagement 

Good teaching 


Questions 10, 16, 15, 18: 

Questions 20, 5, 19, 9: 


10 I enjoy doing the work for this course 

20 Staff put a lot of time into commenting 


16 There is a good balance between theory 

on my work 


and practice 

5 Teaching staff normally give me 


15 I feel I can actively participate in my 

helpful feedback on how I’m going 


classes 

19 Staff make a real effort to understand 


18 I can see how I can use what I am 

any difficulties I might be having with 


learning in this course in my career 

my work 


Factor accounts for 18% of variance 

9 Teaching staff motivate me to do my 
best work 

Factor accounts for 16% of variance 


Despite their relatively high and consistent GTS means, a similarly constituted good teaching 
factor emerged prominently from the analysis in both disciplines (though it was slightly more 
salient in communications, where it was the first factor, than in marketing, where it was the 
second factor). There were, however, substantial differences between these disciplines in the 
structure of the other leading factor. Looking at the items comprising this factor suggests that 
in marketing, there was variation in students’ experience of their subjects as purposeful and 
well-organised; whereas in communications, there was variation in their personal sense of 
meaningful engagement. 

This sensitivity to differences in students’ experience is a key feature of the discipline 
level factor analyses. The particular combination of items constituting a given factor suggests 
a distinctive perspective on the meaning of the items. For example, responses to items about 
the organisation of the subject (questions 1, 2, 3 and 4) were frequently correlated. In some 
disciplines (for instance Accounting and Radiation Science) these items also correlated with 
responses to question 18 (on the usefulness of the subject), suggesting that the dimension 
related to whether the subject was seen as purposeful. In other disciplines (for example 
Nursing and Economics), they correlated with items on the requirements and outcomes of the 
subject (questions 7, 8 and 6), suggesting that the dimension related to the subject being seen 
as fit for purpose. 

In addition to the diversity of the dimensions identified by the factor analysis, there 
was diversity in the salience of particular items within a factor. For example, there were 
several disciplines where question 19 (understanding difficulties) was the defining item on the 
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good teaching dimension. This is consistent with early CEQ results. However, question 20 
(staff time on comment) was most frequently the defining item for this dimension. 
Historically, the graduate CEQ has shown a relatively low level of agreement with question 
20, and the university has been emphasising the importance of providing feedback to students; 
it seems likely that the diversity of student experience of feedback has increased as a result. 

Reporting Results 

Once the factors had been identified, the items contributing to each factor were 
grouped together and reports prepared for each discipline showing the items most closely 
associated with the ‘good teaching’ factor. A sample image from the report is shown at Figure 
2 . 



Figure 2 

Items associated with the Good Teaching dimension. 

Note'. The numbers on the target are CES question numbers: the question closest to the bullseye is the one 
loading most heavily on the Good Teaching factor. 


Additionally, each report includes a graph showing the level of agreement with each item and 
a table with the factor structure for the discipline. 

The factor structure is also being used as the starting point for further work comparing 
student responses across subjects within the discipline: the project team has prepared graphs 
using error bars to show how students respond to associated items in different subjects. 

However, communicating the results has been problematic. For staff unfamiliar with 
factor analysis, it is not obvious how to interpret an item that has the highest weighting on a 
factor, or how to interpret differences between factors. 

The complexity of the factor analysis approach makes it particularly important to 
present the results using different perspectives. How might the key results of this work be 
presented so that the underlying logic is clear to a non-mathematician? 

Three concepts seem to be fundamental. 
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1. The highest loading factor is the one which captures the greatest variability. 

This implies that students give a wide range of responses to the items in the factor. If their 
responses were alike there would be no variability to capture. 

Factors are calculated to have a mean of zero and a standard deviation of 1, so factor 
scores per se cannot be used to demonstrate the variability of the underlying data. However, 
one would expect subject-level factor scores to vary and therefore to provide data illustrating 
the diversity within the discipline. Figure 3 depicts means and 95% error bars for subjects in 
the economics discipline (N = 1207) (identifying detail removed). 



Figure 3 

Error bars for Good Teaching factor scores, economics subjects. 


Subject 


As Figure 1 showed, Economics had a relatively low Good Teaching scale mean, of 
just over 20. Figure 3 shows that (as expected) the subjects within Economics are not all al ik e. 
As indicated by the 95% error bars, student responses are consistent within subjects — with 
the obvious exception of subject L, which had responses from only six students. 

The circles in the error bars show the mean factor score for each subject. There are 
clearly significant differences between subjects, with some being rated much more positively 
than others. A couple of subjects (D and N) have a mean factor score nearly half a standard 
deviation above the mean, whereas two other subjects (C and H) have a mean score nearly 
half a standard deviation below the mean. D and N may offer examples of good practice; in C 
and H there appears to be considerable room for improvement. Hence analysis of factor (and 
item) scores by subject provides a useful starting point for further discussion. 

2. The highest loading item on a factor reflects the variation in the other items. 

This implies that there is variation in the responses to the highest loading item. It may 
be thought that the highest loading item on a factor will be one where there is strongest 
disagreement — the item which most needs fixing. This is not so. 
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This can be seen by looking at the distribution of responses to question 20 (teachers’ 
time commenting on student work). As already mentioned, this was the defining item for the 
Good Teaching factor in many disciplines, including all the example disciplines shown in 
Figure 1. Although students were, as expected, much more positive in arts and social science 
disciplines, in all the disciplines there was a broad range of responses. 



Figure 4 

Distribution of responses to Q20: Example disciplines. 

The defining item points to potential for examining both where students’ experiences 
are positive and where they might be improved. In the knowledge that this item varies 
significantly within the discipline, it is clearly valuable to develop more fine-grained analysis 
looking at the experiences of different cohorts (international students, students articulating 
from TAFE, commencing and returning students). This work is being undertaken as part of 
the project. 

3. Students respond consistently to the different items in the factor 

An individual’s response on one item in the factor is likely to be similar to their 
response on the other items. A student who disagrees with one item will be likely to disagree 
with the other items; a student who agrees with one item will be likely to agree with the other 
items. Hence, strongly disagreeing with an item in the factor will be associated with a low 
factor score and strongly agreeing will be associated with a high factor score. 

Data from the project illustrate this. The graph below again draws on data from the 
economics factor analysis. It shows the distribution of Good Teaching factor scores according 
to the response the students gave to question 20 (time on comments). 
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Figure 5 

Error bars for Good Teaching factor scores, economics discipline, by response to Q20. 

It can be seen that, within the discipline, students’ experiences on this item are consistent with 
their experience of the other items included in the factor. The factor points to associated 
practices which can be considered as a set. 

Discussion 

Using the discipline-level factor analysis provides more information about student 
experience than is visible from scanning item level agreements and scale means. 

Applying this approach to Marketing, for example, it will be recalled that Factor 1 
comprised items that seem to relate to students’ experiences of order and purpose in the 
subjects being evaluated (see Table 2: extract below). 

Extract from Table 2 

Marketing 

N = 1451 _ 

Factor 1: Organised and purposeful 
Questions 3 4 2 1 17: 

3 This course is well organised 

4 The teaching staff are extremely good 
at explaining things 

2 I am learning what I expected to in this 

course 

1 The learning objectives in this course 

are clear to me 

17 The teaching staff work hard to make 
this course interesting 
Factor accounts for 19% of variance 

Following through on the theoretical discussion, we can say first that this is the dimension of 
student experience where responses from marketing students show most variation. Hence we 
know that on this dimension marketing subjects include both good practice and opportunities 
for improvement. Some students must have indicated that they were clear about what they are 
doing, that it matched their expectations, and that they understood the work; other students (or 
the same students in a different subject) must have indicated that they felt muddled about both 
the purpose and the content of the subject. 
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An obvious next step is for staff to explore these findings. Where are the difficulties 
students experience? Are there particular groups of students who find the marketing subjects 
confusing? Or are there particular subjects (perhaps problem-based subjects) where students 
are generally confused? Conversely: are there groups of students who are very clear about 
what they are doing and why? Or are there particular subjects that are generally agreed to be 
organised and purposeful? Answering these questions (via further data analysis or via direct 
discussion with students) will identify both opportunities for improvement, and exemplars of 
good practice within the discipline. 

Secondly, the factor analysis tells us that, from the students’ perspective, the items 
within this factor were related. Perceptions of order and purpose constituted a consistent 
dimension of the students’ experience. This gives more to work with than the results for each 
item individually. We can conclude that once staff know where improvement is needed, it will 
be useful for them to make a concerted effort to improve and align communication with 
students about the intentions, outcomes and content of the subjects that are causing difficulty. 

In conclusion 

This project has considerable potential for enhancing the usefulness of subject survey 
data. The results can be used to: 

• identify distinct dimensions of students’ experience 

• explore differences in students’ experience within a discipline 

• identify subjects within the discipline which are exemplars of effective practice on a 
particular dimension. 

The disciplinary starting point focuses attention on the student’s experience, rather than the 
teacher’s performance. The factor analysis enables the identification of areas where students’ 
experiences vary, so that within the body of practice in the discipline there is clearly both a 
need for improvement and positive exemplars. With discussion grounded at discipline and 
subject level, it should be possible to explore contextual and resource issues impacting on 
student experience, along the lines suggested by Knight and Trawler (2000). Used in this 
way, these analyses have the potential to provide a real jumping off point for change. 
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Attachment 1 

Illustrative means for CEQ Good Teaching, 2004-2007 



Visual Arts and Crafts 

— Marketing 

Economics 

—X— Communications and Media Studies 

)l< Civil Engineering 

— 1 •— Accounting 






















